openSUSE:Packaging Go
Build Service Tutorial 路 Tips & Tricks 路 Cross Distribution Howto 路 Packaging checks
Desktop menu categories 路 RPM Macros 路 Scriptlets 路 Init scripts 路 How to write good changes
About golang
Go (or golang, more search engine friendly) is an expressive, concurrent, garbage collected systems programming language that is type safe and memory safe. It has pointers but no pointer arithmetic. Go has fast builds, clean syntax, garbage collection, methods for any type, and run-time reflection. It feels like a dynamic language but has the speed and safety of a static language.
Naming scheme
Please note that golang packages follow a strict naming convention, with exceptions for convenience.
Golang packages' names begin with a prefix "golang-", follow a string modified from $IMPORTPATH, which is the "url" (actually it's a path) when you `go get`. Technically $IMPORTPATH looks like the "include <xxx.h>" in C, "require 'xx'" in Ruby or "from package import function" in Python, but in a format "import golang.org/google/api". In this case "golang.org/google/api" is the $IMPORTPATH.
It can be taken as three parts: vendor => golang.org, author => google, pkg => api. Here's how we modify it to the following string in golang packages' names:
vendor: usually we use the common known name of the vendor, eg. "github.com" will be "github" and "code.google.com/p/" will be "googlecode". When there's no common known name for a vendor, eg. "gopkg.in", we just skip the "dot" in the vendor url, that is "gopkgin".
author: usually this part is unique enough, most of them are people names, so we just keep it as it is. And it can also be skipped in some cases. eg: "code.google.com/p/xxx", there's no author in the importpath at all (not the "p"). But in other cases it must be explicitly kept, eg "golang.org/x/net", the "x" here means that it's some kind of "standard libraries", so it must be kept. And the "nsf" in "github.com/nsf/gocode" can't be skipped at all.
pkg: usually we just keep it as it is. But when there's a string "go" somewhere in the upstream package name:
- go-XX: skip the "go", we don't want to see package name like "golang-github-xxx-go-xxx", the "golang-" prefix indentifies it already
- XX.go: this is old history back to Go1 era. just skip the "go"
- goXX: keep the "go" because it is "in" the package name, not prefix or suffix.
Versioning scheme
Most golang packages did not follow a strict version scheme yet. You should use the date when you retrieved the upstream (e.g. via git/mercurial/etc. checkout) plus the VCS used, like 0.0.0+gitYYYYMMDD, eg 0.0.0+git20150726, or you can let our Build Service do it for you via a _service file. If the upstream maintainers decide to start using a version scheme, this ensures easy package upgrades in the future.
Development repository & process
Golang packages for openSUSE are developed at devel:languages:go repository in the openSUSE Build Service.
If you want to submit a package to this repository, please:
- branch it (easy way: visit this repository and click "branch" button for any existing package inside this repository)
- copy your package from your home repository to the branched repository
- build
- submit
This will ensure that your package will build for this repository.
Things you should know before packaging
Not all packages will be accepted
Not all packages will find their way into the devel:languages:go project. Packages that will be accepted include binary packages (e.g. clair or packer) and their direct and indirect dependencies. All other packages will most likely not be added. We recommend using `go get` for such packages.
Golang do have dependencies
Golang provides a method called "go get", which is commonly used by upstream. It will fetch the source code of the package plus source codes for all the dependencies, build them in /tmp, and install the compiled codes in ~/go/pkg/linux_${go_arch}/. (a combination of download, `go build` and `go install`.)
So just like other new scripting languages, e.g. Nodejs, it will confuse new packagers with dependencies. Some may think this package has no other dependencies except the "go" main package, run into this mess, and find it has about 30+ dependencies to package first. Please check your ~/go directory first before asserting that this package has no other dependencies and running into chaos.
Where will Golang find dependencies?
Golang will find dependencies from $GOROOT (where go itself installs) and $GOPATH (where go packages installs). You can run:
go env
to find out yours. Basically, inside the build infrastructure, it will be:
/usr/lib(64)/go/pkg/linux_%{go_arch}/ ($GOROOT)
/usr/lib(64)/go/contrib/pkg/linux_%{go_arch}/ ($GOPATH)
/home/abuild/rpmbuild/BUILD/go/pkg/linux_%{go_arch}/ ($GOPATH)
Within plain environments, it would be:
/usr/lib(64)/go/pkg/linux_%{go_arch}/ ($GOROOT)
/usr/lib(64)/go/contrib/pkg/linux_%{go_arch}/ ($GOPATH)
~/go/pkg/linux_%{go_arch}/ ($GOPATH)
But in the build log, it will be different because the compilers of go (such things like 6g 6l...) will:
1st: try to find the compiled dependencies in the above path
2nd: try to find source code of the dependencies in the paths mentioned below, compile them, install them to the $GOPATH of the user operating go, then use them as dependencies.
Within the build infrastructure:
/usr/lib64/go/src/pkg/$IMPORTPATH ($GOROOT) /home/abuild/rpmbuild/BUILD/go/src/$IMPORTPATH ($GOPATH) /usr/lib64/go/contrib/src/$IMPORTPATH ($GOPATH)
Within plain environments:
/usr/lib64/go/src/pkg/$IMPORTPATH ($GOROOT) ~/go/src/$IMPORTPATH ($GOPATH) /usr/lib64/go/contrib/src/$IMPORTPATH ($GOPATH)
3rd (only able in `go get` mode): download source code to the $GOPATH of the user operating go and repeat the 2nd step.
So if OBS gives you an error, it usually happens at the 2nd step because you carelessly used the `go get` method to replace the default %gobuild macro, which will show such errors like "can't find git" or "network not reachable".
The Go layout
The standard directory layout for go looks like this:
go
.. pkg
.. linux_%{go_arch}
.. $IMPORTPATHs for packages (compiled codes reside here)
.. src
.. pkg
.. $IMPORTPATHs for packages (source codes reside here)
.. contrib
.. pkg (same as above)
.. src (same as above)
The contrib directory is used for 3rd-party implementations (usually our packages). But in ~/go, Go will place compiled codes in pkg and source code into src directly (no contrib directory at all).
Example template
You can use the following spec file templates. Make sure to replace $VARIABLES with appropriate values.
Go modules based
Most golang based applications will probably use go modules to manage their dependencies nowadays. This means we can use this to collect all the dependencies into a so called vendor tarballs. To help with simplifying this step the openSUSE project has created obs-service-go_modules. With a _service file in our package we can set this up for us:
<services> <service name="go_modules" mode="disabled"> <param name="compression">zst</param> </service> </services>
Running osc service runall now should create the vendor.tar.zst for us. For all supported parameters check
the go_modules documentation
# # spec file for <package name> # # Copyright (c) 2023 SUSE LLC # # All modifications and additions to the file contributed by third parties # remain the property of their copyright owners, unless otherwise agreed # upon. The license for this file, and modifications and additions to the # file, is the same license as for the pristine package itself (unless the # license for the pristine package is not an Open Source License, in which # case the license is the MIT License). An "Open Source License" is a # license that conforms to the Open Source Definition (Version 1.9) # published by the Open Source Initiative. # Please submit bugfixes or comments via https://bugs.opensuse.org/ # Name: name Version: 0.0.1 Release: 0 Summary: <summary> License: License-2-Clause Group: Productivity/Networking/Instant Messenger URL: <package homepage url> Source: https://upstream/path/to/%{name}-%{version}.tar.gz Source1: vendor.tar.zst BuildRequires: golang-packaging BuildRequires: zstd %description Description. %prep %autosetup -p1 -a1 %build go build \ -mod=vendor \ -buildmode=pie %install install -D -m0755 %{name} %{buildroot}%{_bindir}/%{name} install -D -m0644 man/%{name}.1 %{buildroot}%{_mandir}/man1/%{name}.1 %files %license LICENSE %doc README.md %{_bindir}/%{name} %{_mandir}/man1/%{name}.1%{?ext_man} %changelog
Old style golang applications
For golang software which does not support you can use the golang macros to simplify building. An example spec file is shown below.
# # Copyright (c) 2021 SUSE LLC # # All modifications and additions to the file contributed by third parties # remain the property of their copyright owners, unless otherwise agreed # upon. The license for this file, and modifications and additions to the # file, is the same license as for the pristine package itself (unless the # license for the pristine package is not an Open Source License, in which # case the license is the MIT License). An "Open Source License" is a # license that conforms to the Open Source Definition (Version 1.9) # published by the Open Source Initiative. # Please submit bugfixes or comments via https://bugs.opensuse.org/ # %global provider_prefix github.com/example/repo %global import_path %{provider_prefix} Name: golang-$STRING Version: 0.0.0+git20140916 Release: 0 License: $LICENSE Summary: $SUMMARY Url: $HOMEPAGE Group: Development/Languages/Other Source: $EXACT_UPSTREAM_NAME-%{version}.tar.xz BuildRequires: golang-packaging BuildRequires: xxx-devel BuildRequires: xz Requires: xxx-devel %{go_provides} %description $DESCRIPTION %prep %autosetup -p1 -n $EXACT_UPSTREAM_NAME-%{version} %build %goprep %{import_path} %gobuild . %install %goinstall %gosrc %gofilelist %check %gotest %{import_path} %files -f file.lst %license LICENSE %doc README %changelog
The complicated explanation
Macros
You can look up the provided macros in the file /usr/lib/rpm/macros.d/macros.go.
Some common macros are explained below (But not all of the macros, because some macros like %go_exclusivearch are for internal use and have already been included in other macros).
NOTE: Some macros can not be quoted
Usually, you already have had an idea that %make equals to %{make}, just like shell scripts.
This rule applies to almost every aspect of RPM packaging. But in golang packaging you have to eliminate such careless ideas from your mind.
As I know, %goprep, %gofix, %gobuild, %goinstall, %gosrc, %gotest are macros that can not be quoted because they allow arguments. So please double-check after you run "spec-cleaner -i xxx.spec", which will quote such macros.
%go_ver
It will return golang's version like 1.4.2
%go_arch
It will return golang's architecture: i386 or amd64.
%go_nostrip
You should not use this. It will disable stripping of the binaries. Long ago using strip on go compiled binaries caused bugs, but that seems to have been fixed.
%go_requires
Removed in golang-packaging v15.0.0.
It will only Requires go main package for your package.
%go_provides
It will add Provides %{name}-devel and %{name}-devel-static for your package.
Unlike C/C++ packages, Golang packages do not have header files. They are statically built so the main package is also the devel package.
%go_recommends (Removed)
Removed in golang-packaging v15.0.0.
It will add Recommends %{name}-doc sub-package for your package.
Unlike the common RPM packaging, %{name}-doc is actually the source package in golang, instead of documentation packages.
%godoc_package (Removed>
Removed in golang-packaging v15.0.0.
Like %lang_package, it will create a %{name}-doc sub-package in specfile.
%go_contribdir
go binaries/static libraries will be installed into this directory:
%{_libdir}/go/contrib/pkg/linux_%{go_arch}
%go_contribsrcdir
go sources will be installed into this directory:
%{_datadir}/go/contrib/src/pkg
%goprep
This macro will prepare the build environment for golang.
You have to give an argument $IMPORTPATH to it. If you have seen some golang specfiles, you will find something like this:
%goprep gopkg.in/qml.v1 %goprep github.com/howeyc/fsnotify
These gopkg.in/qml.v1 or github.com/howeyc/fsnotify are actually not URL but PATH, although http://gopkg.in/qml.v1 is reachable (Actually it's the homepage for this library).
%goprep will create a %{_builddir}/go/src/gopkg.in/qml.v1 directory.
Then what's the purpose for this $IMPORTPATH and how to find it?
It is used for importing. eg. in golang code:
import (
qml gopkg.in/qml.v1
)
And golang will find such library in
%{go_contribsrcdir}
~/go/pkg/linux_amd64/
/home/abuild/rpmbuild/BUILD/go/pkg/linux_amd64/ (This only exists in OBS build environment, that is, ~/rpmbuild/BUILD/go/pkg/linux_amd64/ will not count unless you're building inside it)
And you can see the examples provided by upstream to easily find the $IMPORTPATH in source tarball.
%gobuild
It actually builds and installs the compiled golang codes into %{_builddir}.
By default it will build everything inside %{_builddir}/go/src/$IMPORTPATH, if you want to selectively build some sub-directories inside $IMPORTPATH, please install "go" package and read /usr/lib/rpm/macros.d/macros.go for $IMPORTPATH_MODIFIER arguments' usage.
%goinstall
It will copy the compiled codes from %{_builddir} to %{buildroot} for RPM package creation.
%gosrc
It will scan the source directory and copy all *.go, *.s, and *.h files to %{go_contribsrcdir}.
%gofilelist
It will scan %{go_contribsrcdir} for files and directories, and generate a file (file.lst) listing each.
This file can then be used for the %files macro, see here.
%godoc
No-op in golang-packaging v15.0.0.
It will scan the source directory and copy all *.go files to %{go_contribsrcdir} to create a %{name}-doc package, just like %find_lang but without arguments.
%gotest importpath
It will test all golang codes inside importpath directory.
The changes of BUILD directory layouts
There're more than one derivatives for this case:
- If you got a "File or directory not found" error and you want to do a "ls -l" in sections like %install.
- If you want to selectively put files into %{name}-doc packages instead of putting everything upstream provides. This usually involves file addition/deletions, which further suppose you know the directory layout of the folder you are operating under.
Why?
Golang is a structure sensitive language. It needs such layouts to build and place build results.
Layout after %goprep
After running %goprep, an additional Go-specific directory structure is created.
All files from BUILD/%{name}-%{version} are then be copied to the new directory BUILD/go/src/$IMPORTPATH.
Although RPM will still cd into BUILD/%{name}-%{version} when reaching the sections %build or %install, this new directory structure is necessary for the package itself and its dependencies.
It will also be set in the GOPATH environment variable which in turn is used when running %gobuild.
Layout after %gobuild
While after running %gobuild, you package will be installed as
BUILD/go/pkg/linux_%{go_arch}/xxx/xxx/%{name}.a
or
BUILD/go/bin/%{name}
Layout after %goinstall
After running %goinstall, the package will be installed to
BUILD/usr/bin/
Static libraries (*.a files) are not installed, and therefore won't be part of the shipped package.
Layout after %gosrc
After running %gosrc, your package source will be installed into
BUILDROOT/%{_datadir}/go/contrib/src/pkg/xxx/xxx/%{name}/
It will copy *.go files, as well as *.s and *.h files. Should any other files be required, these have to be copied manually into this path.
There is no %{name} directory in %{go_contribdir}
From previous chapters we know xxx/xxx/%{name} is not a URL but $IMPORTPATH.
So some people may try to "pushd BUILDROOT/%{go_contribdir}/xxx/xxx/%{name}" and then do "ls -l" or "find" to see what's wrong.
But there'll be no such directory in BUILDROOT/%{go_contribdir}/. Actually only xxx/xxx exists, %{name}.a is actually a static compiled file. And if the source has subdirectories, they will be compiled as xxx/xxx/%{name}/<subdirectory-name>.a or xxx/xxx/%{name}/<subdirectory-name>/<some-other-name>.a. But anyway:
xxx/xxx/%{name}/*.go will be compiled to xxx/xxx/%{name}.a
That is, you will never see anything related even if you can "cd" into the %{name} directory after build.
NOTE: this only applies to BUILDROOT/%{go_contribdir}, BUILDROOT/%{go_contribsrcdir} still has the %{name} directory containing the *.go files.
Dependencies (Requires)
Golang packages are "static" builds, so shlib resolving tool (/sbin/ldconfig) does not work.
Easy cheat sheet
Remember this: Your BuildRequires is also your Requires.
It may sound weird, but to some extent Golang is like Python. You can run Python codes in binary-compiled code (*.pyc) or source code (*.py), and you can run Golang codes in binary-compiled code (*.a) or source code (*.go) too. (But unlike Python, "go run" can't run compiled codes.)
So what you need in your build environment will be what you will need for you runtime environment.
Don't be fooled by the ".a" suffix. In C this means static, you don't need to install shared lib dependencies for it then. But in Golang, it will not absorb all the dependencies. It will not absorb the C dependency codes like those from GTK either.
Learn the hard way
"Pure" Golang packages
If you skip the essential Requires (usually are those essential to build the package, eg. go-log4go for lime), it will cause an error when the user runs the package:
test.go:2:8: cannot find package "code.google.com/p/log4go" in any of:
/usr/lib64/go/src/pkg/code.google.com/p/log4go (from $GOROOT)
/home/marguerite/go/src/code.google.com/p/log4go (from $GOPATH)
/usr/lib64/go/contrib/src/code.google.com/p/log4go
This can be fixed by providing Requires and BuildRequires:
Requires: golang(code.google.com/p/log4go) BuildRequires: golang(code.google.com/p/log4go)
Some errors however might seem a bit more complicated to solve:
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: ([]interface {}) (0x64c540,0xc208045cc0)
[signal 0xb code=0x1 addr=0x0 pc=0x4165ba]
...
[ follwing with a very looong "goroutine" ]
...
In this case, simple tests on your local can help locating the problem:
- For a library: You can just download the source from upstream and run
go run xxx.goto see where it fails. - For an application: Just uninstall all Golang packages except the "go" base package and run your application. This way you can catch all the runtime dependencies.
Golang wrapper packages for C/C++
Golang wrappers for C/C++ libraries, like go-gozmq for zeromq, will use pkg-config to find its runtime C/C++ dependencies, which means if you do not have "pkg-config" and "zeromq-devel" (which provides /usr/lib64/pkgconfig/zeromq.pc) installed, gozmq will not able to detect libzmq4 library even if you have it installed. When you run examples, it will complain:
$ go run client.go # command-line-arguments /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -lzmq collect2: error: ld returned 1 exit status
For such Golang wrappers it may therefore be necessary to install a *-devel package in order for the package to build.
Common Golang errors
This section collects some common Golang errors for those packagers who know little about Golang.
undefined reference to SOME_C_FUNCTION
Golang can call functions written in pure C with a tool named cgo. If you see
/* #include <stdio.h> */ import "C"
in a .go file, this file calls some C functions. As a qualified packager, we all more or less have some ideas about CFLAGS and LDFLAGS, thus know "undefined reference" error means the compiler couldn't find related header files or *FLAGS in C. But how could we fix such errors in Golang? Here it is:
- fix for headers are pretty easy.
- Here's how to fix CFLAGS and LDFLAGS:
/* #include <stdio.h> #cgo pkgconfig: geoip #cgo LDFLAGS: -lGeoIP #cgo CFLAGS: -xxxxxx /*
main redeclared in this block
Golang is a structure sensitive language. If a folder contains multiple files and those files have more than one "main" function, this error triggers. But some old go projects (eg. since Golang 1.1) still didn't put their examples files in separate folders like "examples". We know examples are some small, independent demos relying on this project, which must have a separate "main" function. so the fix is easy, just "mkdir -p" an "examples" folder and put all such files into it in %prep section in specfile (see #Layout after %goprep chapter).
object is [linux amd64 go1.3 X:precisestack] expected [linux amd64 go1.3.1 X:precisestack]
This error means your package is built against go 1.3, but your current Golang environment is go 1.3.1. Usually a rebuild will solve the problem.
stat "bitbucket.org/taruti/ssh.go" No such file or directory
It's because "*.go" is no longer allowable for import path in new versions of Go.
So in this case, you need to change
%goprep bitbucket.org/taruti/ssh.go
to
%goprep bitbucket.org/taruti/ssh-go
and mention that in .changes file so developers using our repository will know the change.
But usually it indicates this project is too old and you should find newer implementations if upstream didn't adapt to the change themselves. In the above case, you should use "code.google.com/p/go.crypto/ssh" which is newer and official.
Packaging tricks for Go
"tags" in go get
Sometimes upstream e.g. gozmq told you to "go get -tags xxx $IMPORTPATH" to get a specific version , and they put files for all tags together in one repository. What's worse, newer tags like "zmq_4_x" needs files from older tags like "zmq_3_x" and "zmq_2_2" to build (Maybe that's why upstream didn't split files into different git branches). Plus the "*_test.go" files, they totally get us confused about which file can be deleted for the build of a specific version.
So we can't simply deleted "zmq_3_x" files for "zmq_4_x" builds, and considering the "*_test.go" files, it'll be a total mess for us to use "%{?suse_version}" to delete files (illogic, changeable all the time). All in all, "go get -tags" is the perfect, most simple, and (maybe) only possible way.
In this case, we need to manually "construct" a "go get" to replace %gobuild. As #Where will Golang find dependencies? shows, "go get" will try to find sources in $GOPATH before actually downloading online (the later will certainly break because OBS build VM doesn't have any network), it indicates that if we put sources at the right place (%goprep) and export the correct envrionment variables, "go get" is usuable on OBS. So we can change commands in specfiles like this:
%build %goprep github.com/alecthomas/gozmq %gobuild
%install %goinstall
to
%build
%goprep github.com/alecthomas/gozmq
# manual build
export IMPORTPATH="github.com/alecthomas/gozmq"
export BUILDFLAGS="-s -v -p 4 -x"
export GOPATH=%{_builddir}/go:%{_libdir}/go/contrib
%if 0%{?suse_version} >= 1230
%if 0%{?suse_version} > 1315
go get --tags zmq_4_x $BUILDFLAGS $IMPORTPATH
%else
go get --tags zmq_3_x $BUILDFLAGS $IMPORTPATH
%endif
%else
go get --tags zmq_2_1 $BUILDFLAGS $IMPORTPATH
%endif
%install %goinstall
$GOTOOLDIR is not rewritable
Sometimes when building Go official packages e.g. go.tools, some binraries like "godoc" will force to install themselves into %{_libdir}/go/pkg/tool/linux_%{go_arch}, even if you have recontructed the %gobuild macro and exported GOTOOLDIR to %{_builddir}/go/pkg/tool/linux_%{go_arch}.
This is intentional because Go upstream doesn't want official tools like godoc to be separately updated without the rest of go tools. But as go.tools is separately built from go main package, such decision will trigger "open /usr/lib64/go/pkg/tool/linux_amd64: Permission Denied" error on OBS.
We patched go main package and provided a macro called "GOROOT_TARGET". If you:
export GOROOT_TARGET="%{buildroot}%{_libdir}/go"
before %gobuild, you can get this workaround.